Gender Differences 1 Running head: GENDER DIFFERENCES IN SEARCHING THE WEB Gender Differences in Patterns of Searching the Web

There has been a national call for increased use of computers and technology in schools. Currently, however, little is known about how students use and learn from these technologies. This study explores how eighth-grade students use the Web to search for, browse, and find information in response to a specific prompt (how mosquitoes find their prey). A previous analysis (Roy, Taylor & Chi, in press) found that boys performed significantly better on gaining target-specific (information directly related to the prompt) and target-related (information related to mosquitoes in general) knowledge than girls. The current paper explores this difference further by examining how students searched the Web for information. Each student’s search behavior was diagramed out and a series of six different “search moves” were derived. Statistical analysis of these search variables revealed that boys tended to employ a different search pattern from girls and that this variation in search behavior was related to the pattern of performance outcomes. Gender Differences 3 Gender Differences in Patterns of Searching the Web The World Wide Web (Web) is a large distributed hypermedia repository of information where people can search for information, navigate through links, and view pages of documents through browsers. Web users are faced with finding their way through this rich but unstructured information world. Increasingly, students are using this incredible resource from both school and home as a main avenue for finding information for their research projects and papers (Wallance, Kupperman, Krajcik & Soloway, 2000). Thus, students need to navigate this complex world of information effectively. But what are these navigational skills precisely, and are there important individual differences in the ways that students search for information on the Web? Previous research has shown that students vary widely in their ability to find and retrieve information in loosely structured information environments (Chang & McDaniel, 1995). Some factors that predict search success in such environments include level of domain knowledge and search expertise (MaKinster, Beghetto & Plucker, in press; McGregor, 1999), ability (Chang & McDaniel, 1995), gender (ChanLin, 1999), learner control (Diddon & Gabbard, 1998; McGreggor, 1999), learner style (Shute, 1993), and interest (Tobias, 1994). In general, more experienced, knowledgeable, interested, male users who are active learners and oriented towards an internal locus of control are associated with being successful in such environments. While several studies have investigated search behavior in closed hypermedia systems, it has been suggested (Hargittai, 2002) that finding information on the Web may be an even more complex search task in that the Web allows many different search strategies. Students can have difficulty finding useful, relevant, and accurate information on the Web. For example, students often lack the skill and resources to access and manage information intelligently (Salomon, 1998). Moreover, according to Wallace et al. (2000) students use the Web simplistically in that they tend to engage in “information gathering” (simply searching for and retrieving relevant Gender Differences 4 information as an end in itself) rather than in“information seeking” (the iterative and recursive process of finding and using relevant information to refine the search goal until it is met). Such findings suggest that the Web, in and of itself, might not be an appropriate resource for supporting such open ended learning tasks. A similar conclusion was reached by Vansickle and Monaco (2000) in their study of tenth graders use of the Web. They reported that students did not demonstrate much search sophistication. For example, students failed to see connections between the amount and kinds of information they retrieved and the way they searched for information. Search skills definitely improve with domain knowledge. To find information about a specific science concept, searchers with more sophisticated domain knowledge demonstrated greater understanding of the organization of scientific content, greater ability to evaluate Web page descriptors and content, and greater facility formulating appropriate keywords than searchers with less domain knowledge. Students with more impoverished domain knowledge had greater difficulty formulating effective search keywords, selecting appropriate sites or documents to pursue, and demonstrated a greater tendency to become distracted during the search process (Makinster et al., in press). This clearly suggests that search skills depend on content knowledge. But how do students fare when they are asked to use the Web to search for information about a topic for which they have little content knowledge? This would seem to be an important question to address since this is the type of task students are more routinely asked to perform. One individual difference that may be significantly related to successful search behavior is gender. The literature on gender differences in academic achievement in general is extensive. It is widely accepted that females score higher on average than males on tests of verbal abilities (Hyde & Linn, 1988), while males score higher on average than females on tests of mathematical Gender Differences 5 ability (Hyde, Fennema, & Lamon, 1990) and spatial abilities (Linn & Peterson, 1986). One might also expect gender to contribute to student’s ability to successfully search and navigate information on the Web, since there is evidence suggesting that girls lag behind boys in the degree to which they are experienced with and motivated by technology (Ligth et al., 2000; Schumacher & Morahan-Martin, 2001) and their attitude towards computers and experience using the Web (Leong & Hawamdeh, 1999). Such findings suggest that boys might out perform girls in their ability to search the Web since they are more experienced users and have more positive attitudes towards technology in general. But how does gender relate to search performance more specifically? Large, Beheshti, and Rahman (2002) investigated how same-sex collaborative groups of grade six students searched the Web for information to complete a class assignment. They found boys to be generally more active in the way they searched the Web, formulating more queries, clicking on more links per minute, and following up on more hits. On the other hand, girls spent more time reading documents. Although Large et al. (2002) quantified and compared aspects of search behavior across gender, they did not relate these behaviors to any measure of search success. Therefore it is not known if such differences are important in distinguishing successful from unsuccessful search patterns. Furthermore, it is possible that there are important differences between boys and girls not only in terms of the distribution of individual search behaviors, but also in terms of how those behaviors are orchestrated into a coherent search pattern. The current paper explores gender differences in how middle school students search the Web to find information in response to a specific prompt for a case where students possess little domain knowledge. That is, when students are unfamiliar with a topic, are boys and girls equally successful at locating information using the Web? What search patterns do they demonstrate? How do such search patterns relate to success in finding and using the targeted information? Gender Differences 6 The data reported here is part of a larger study comparing the ability of students to search for information on the Web verses in the school Library. In an earlier paper (Roy, Taylor & Chi, in press) we reported that boys and girls did not differ in their ability to search for, locate, and summarize information when using the school Library. However, when using the Web for an identical search task, boys performed significantly better than girls on both target-specific information (i.e., information they were specifically prompted to find) and target-related information (i.e., information that was related to the target topic, but was not the specifically prompted to find). This paper further explores this gender difference in the Web condition by analyzing student’s patterns of search behavior and by attempting to relate these differences to performance outcomes. If variation in the search patterns employed by boys and girls does explain the observed performance differences, then a careful analysis of these differences should inform the design of tools or scaffolds to support student’s navigation. Method Participants Fourteen eighth grade students from a local Pittsburgh middle school between the ages of 13 and 14 volunteered to participate in this study. Equal numbers of boys and girls participated. Materials The experiment was performed in the school computer laboratory on a Macintosh G3 computer running Internet Explorer as the Web browser and Google as the search engine. Target Domain and Knowledge Assessment. Mosquito behavior (i.e., how mosquitoes find their prey) was chosen as the target topic for several reasons. An informal survey of students prior to the study indicated that the topic was of interest and that most students did not know the answer. Furthermore, the question could be answered within a single academic period and there were sufficient available resources on the Web to answer the question successfully. Gender Differences 7 The materials used to measure student knowledge included (1) an open-ended prompt to assess student’s targetspecific knowledge (In your own words, please describe at least three ways that mosquitoes find their prey.), and (2) a set of 13 short answer questions to assess student’s target-related knowledge. The target-specific prompt was administered to assess both pre-search and post-search target-specific knowledge. The set of target-related questions was used to assess each student’s post-search general knowledge about mosquitoes only. These questions were designed to assess knowledge of facts that students were unlikely to know before their search but were likely to come across while performing their search (e.g., Why are mosquito bites itchy?). We asked these questions to assess general information about mosquitoes that students may have acquired incidentally while searching for information to the targeted prompt. Design and Procedure Each student met with two experimenters (one male and one female) in the school computer laboratory for an academic period (50 minutes) to perform the search task. Initially, each student was given five minutes to respond to the target-specific prompt. Following this pretest of target-specific knowledge, each student was given twenty-five minutes to search the Web for answers to the target-specific prompt. Students were permitted to take notes during their search and to "book mark" up to 30 pages of resources (by selecting “Add to Favorites” under the Favorites menu). Students were told that they would be permitted to refer back to these pages or notes during the posttest section. While each student conducted their search, the experimenters observed and recorded the participant’s search behavior and noted the identity, number, and sequence of all search queries executed and all resources that were examined by the student. The experimenters also recorded any bookmark selections or note taking. This information was used to supplement the computer logs of all Web pages accessed by students. Gender Differences 8 After completing the search, participants had ten minutes to answer the target-specific knowledge question again. Following this posttest, students were given the unannounced targetrelated test to measure incidental facts about mosquitoes that they might have gained during their search. Students were not permitted to use their notes for this assessment. Diagramming Student Search Behavior & Movement Using the field notes along with computer logs of all the Web pages accessed by students during their search, we were able to construct diagrams depicting each student’s search behavior. All search moves were categorized into one of the following four categories of search behavior: (1) submitting a search query in the Google search window; (2) scanning the list of returned document excerpts which contain links to documents; (3) Selecting, opening, and browsing a particular document; and (4) book marking a document location or taking notes. We arranged these behaviors in four columns from left to right. Each new instance of behavior within one of these four state categories was depicted on a new row. Finally, we linked these states with sequenced directed arrows in a manner that reflected the temporal order of movement from one state to another. Figures 1a and b below show small excerpts from two search episodes. These give some flavor of two overall patterns of search behavior that will be elaborated below (i.e. horizontal and vertical search moves). --Insert Figures 1a & 1b here--. Horizontal movement across the search diagram columns represents movement between (1) Submitting a search and Scanning pages describing the search results returned; (2) Scanning pages describing the search results to Browsing an actual document; or (3) Browsing documents to Book Marking or Note Taking (presumably after one has located the sought after information). Each horizontal move across search categories of behavior can be conceptualized as a change in the particular stage of a search episode, from early and more general (moving left Gender Differences 9 wards) to more specific or detailed (moving right wards). For example, the bottom of Figure 1a is an instance of a student searching horizontally. In this instance the student Submits his fourth search query (Mosquito prey), then Scans the first page of returned results (arrow 6), then Opens and Browses a document (arrow 7), and Takes Notes about carbon dioxide and octenal (arrow 8). On the other hand, vertical movement in the search diagrams represent instances where the student pursues a search at a particular stage in a depth-wise manner. In these cases the student moves from (1) search to search; (2) one page search result descriptions to another page; or (3) document to document by clicking on links. We can conceptualize each vertical movement within each search stage as an instance where the student pursues a particular line of search (movement down) or as revisiting information already browsed (movement up). For example, Figure 1b above depicts an instance of vertical search behavior. Specifically, the student moves through a sequence of documents (arrows 17 though 20) by clicking on a series of links. Deriving Measures of Search Behavior. Each subject’s total number of search moves was broken into proportions reflecting changes in search stage (submitting a search query to scanning results; scanning results to browsing documents; and browsing documents to taking notes) and changes in search depth within a stage (movement from search to search, results to results, document to document). Thus search behavior was indexed by six variables, three that reflected state changes in search stage and three that reflected changes in search depth within a particular stage. These variables served as the main quantitative indices for describing search patterns. Results We begin our presentation of results by revisiting the gender differences in performance outcomes that students gained from their Web searches, as reported in a previous paper (Roy et al., in press). In that study, although both boys and girls gained significantly from pre to posttest, Gender Differences 10 boys performed significantly higher than girls on both target-specific (Figure 2a) and targetrelated information measures (Figure 2b). --Insert Figures 2a & 2b here--. We know from the pretest analysis that boys and girls did not differ in their prior knowledge of the target domain. Nor were there differences in terms of familiarity or access to computers, with using Google, or with using the Web. Furthermore, boys and girls did not differ in terms of the total number of search moves, nor in terms of the amount of time spent searching. We turn now to an analysis of Search Behavior in an attempt to explain the learning differences. Overall Search Behavior. The six search variables were analyzed together in a one-way MANOVA with gender as the between-subjects variable. Using a MANOVA enabled us to test the gender difference on the overall pattern of all search variables considered simultaneously. There was a significant multivariate effect for gender (F (6,7) = 12.271, p = .002) indicating that the overall pattern of search behavior was different for boys and girls. Search Stages: Horizontal Moves. On average, boys demonstrated significantly more movement early on in the search phase than girls. Specifically, boys showed a greater tendency to move between Submitting searches and Scanning the results pages of returned by search queries (boys M = .306 and girls M = .101; F (1,12) = 25.528, p < .0005). Boys and girls did not differ from one another on the remaining horizontal move variables. This finding suggests that boys tended to filter information at an earlier stage in the search cycle than girls (Figure 3a). Search Depth: Vertical Moves. On the other hand, for vertical moves (Figure 3b), there is a trend for girls to demonstrate proportionally more vertical movement (down) between documents (boys M = .079 and girls M = .254; F (1,12) = 4.564, p = .054). Such results suggest that girls were much more linear and thorough navigators than boys. --Insert Figures 3a & 3b here--. Gender Differences 11 Relating Search Patterns to Performance Outcomes. We performed a correlation analysis in order to investigate whether the observed gender differences in search behavior predicted the observed knowledge gains. This analysis revealed that target-specific performance gains were significantly and positively related to the proportion of horizontal movement between submitting a search and scanning search results (r = .553, p = .040). Furthermore, there was a trend for a substantial negative relationship between knowledge gains and the proportion of vertical downward movement between documents (r = .485, p = .079). None of the search variables were related to performance on the target-related questions. Relating Search Variables to Knowledge Gains within Gender. Finally, does search behavior relate to knowledge outcomes overall (i.e., independently of gender)? Each subject was categorized as demonstrating either a high or low knowledge gain. A high knowledge gain for girls was defined as a gain of more than 3 correct ideas (i.e., higher than the group mean gain for girls, M = 2.43) , whereas for boys a high knowledge gain was defined as a gain of more than 5 correct ideas (i.e., higher than the group mean gain for boys, M = 4.86 ). Independently, each subject’s proportion of horizontal search movement (between search submitting and result scanning) was compared to the proportion of vertical search movement (between documents) demonstrated. Depending on which proportion was higher, each subject was then categorized as either a horizontal or a vertical searcher. For example, a student whose proportion of vertical moves was .17 and proportion of horizontal moves was .83, was coded as a horizontal searcher. There was a significant association between knowledge gain and the search preference (χ(1, N = 14) = 4.20, p = .040). All students who demonstrated high learning gains also demonstrated a horizontal search preference (6 out of 6), regardless of gender. In fact of the 6 subjects who fell into this cell, 4 were boys and 2 were girls. On the other hand, there was no overall preference for horizontal (N = 4) or vertical (N = 4) search pattern for students who had a low (N = 8) Gender Differences 12 learning gain. Thus, this analysis provides us with converging evidence that learning gains were related to differences in search patterns in a manner that is independent of gender. Discussion The purpose of this study was to investigate how middle school students search the Web to find the answer to a specific question and to relate such patterns to variation in performance outcomes. In a previous study, we observed a gender difference in students’ performance when they were asked to use the Web to perform a well-defined search (Roy, Taylor, & Chi, in press). Specifically, boys demonstrated better performance than girls on both target-specific and for target-related information measures. We were interested in determining if differences in patterns of search behavior could be contributing to this effect. This is an important issue to address since previous studies have suggested that girls may lag behind boys in the degree to which they are experienced with and motivated by new technology (Light et al., 2000; Schumacher & MorahanMartin, 2001). However, it is also possible that the observed gender difference is due to differences in individual search behaviors as the results from Large et al. (2002) suggest; or even in terms of how search behaviors are orchestrated by searchers into search patterns. Although the boys and girls in our study had similar background knowledge about the content domain and similar experience with Web searching, we found two distinct global patterns of search behavior that reliably and independently distinguished girls performance from boys, and high knowledge gain performers from low gain performers. This finding is in agreement with the results of Large et al. (2002). However, our analyses further describe differences in how boys and girls orchestrated individual search behaviors to result in different overall search patterns. Boys had a tendency to oscillate between submitting searches and scanning the document excerpts returned as search results (horizontal searchers). Conversely, girls had a tendency to actually open and browse entire linked Gender Differences 13 documents (vertical searchers) without going through a preliminary filtering step of scanning. Our correlation analysis shows that this pattern of variability was responsible for the different performance outcomes. Gender Differences 14 ReferencesChang, C.K., & McDaniel, E. D. (1995). Information search strategies in looselystructured settings. Journal of Educational Computing Research, 12(1), 95-107.ChanLin, L.J. (1999). Gender differences and the need for visual control. InternationalJournal of Instructional Media, 26(3), 329-335.Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review ofthe quantitative literature on learner comprehension, control, and style. Review of EducationalResearch, 68(3), 322-349.Hargittai, E. (2002). Beyond logs and surveys: In-depth measures of people’s Web useskills. Journal of the American Society for Information Science and Technology, 53(14), 1239-1244.Hyde, J. S., Fennema, E., & Lamon, S. J. (1990). Gender differences in mathematicsperformance: A meta-analysis. Psychological Bulletin, 107, 139-153.Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis.Psychological Bulletin, 104, 53-69.Large, A., Beheshti, J., & Rahman, T. (2002). Gender differences in collaborative Websearching behavior: An elementary school study. Information Processing and Management, 38,427-443.Leong, S.C., & Hawamdeh, S. (1999). Gender and leaning attitudes in using Web-basedscience lessons. Retrrived from http://information.net/ir/5-1/paper66.html.Light, P. Littleton, K., Bale, S., Joiner, R., & Messer, D. (2000). Gender and socialcomparison effects in computer-based problem solving. Learning and Instruction, 10, 483-496.Linn, M. C., & Peterson, A. C. (1986). A meta-analysis of gender differences in spatialability: Implications for mathematics and science achievement. In J. S. Hyde & M.C. Linn Gender Differences 15 (Eds.), The psychology of gender: Advances through meta-analysis. Baltimore: Johns HopkinsUniversity Press.MacGregor, S. K. (1999). Hypermedia navigation profiles: Cognitive characteristics andinformation processing strategies. Journal of Educational Computing Research, 20(2), 189-206.MaKinster, J. G., Beghetto, R. A., & Plucker, J. A. (in press). Why can't I find Newton'sThird Law?: Case studies of students using of the web as a science resource. Journal of ScienceEducation and Technology.Roy, M., Taylor, R., & Chi, M.T.H. (in press). Searching for information on-line and off-line: Gender differences among middle school students. Journal of Educational ComputingResearch.Salomon, G. (1998). Novel constructivist learning environments and novel technologies:Some issues to be concerned with. Research Dialogue in Leaning and Instruction, 1(1), 3-12.Schumacher, P., & Morahan-Martin, J. (2001). Gender, Internet and computer attitudesand experiences. Computers in Human Behavior, 17(1), 95-110.Shute, V. (1993). A comparison of learning environments: All that glitters. In S. LaJoie& S. Derry (Eds.), Computers as Cognitive Tools (pp.47-73). Hillsdale, NJ: LEA.Tobias, S. (1994). Interest, prior knowledge, and learning. Review of EducationalResearch, 64, 37-54.Vansickle, S, & Monaco, M. (2000). Tenth graders’ search knowledge and use of theworld wide Web. Unpublished doctoral dissertation, Georgia State University.Wallace, R., Kupperman, J., Krajcik , J., & Soloway, E. (2000). Science on the Web:Students online in a sixth-grade classroom. The Journal of the Learning Sciences, 9(1), 75-104. Gender Differences 16 Author NoteMarguerite Roy and Michelene T. H. Chi, Learning Research and Development Center,University of Pittsburgh.We would like to acknowledge the contribution of Roger Taylor who assisted in thecollection of the data. In addition, we would like to thank the students and staff who kindlyvolunteered to participate in this study. The authors are grateful for the support from the SpencerFoundation granted to Michelene T. H. Chi.Correspondence concerning this article should be addressed to either Marguerite Roy([email protected]) or Michelene Chi ()[email protected] at 3939 O’Hara Street, Learning Researchand Development Center, University of Pittsburgh, Pittsburgh, PA 15260. Gender Differences 17 Figure CaptionsFigure 1. a) An example of an excerpt from a horizontal search.b) An example of an excerpt from a vertical search.Figure 2. a) Pre and post target-specific knowledge scores by gender.b) Target-related knowledge scores by gender.Figure 3. a) Proportion of horizontal search moves by gender.b) Proportion of vertical search moves by gender. Gender Differences 18